backlog: file #1101, the connscale empty_claims_monotonic SLO reports runner contention as an engine defect - #284
Merged
Conversation
… runner contention as an engine defect The SLO at tests/test_connscale_smoke.py:170 asserts an empty-claim rate PER SECOND, so wall clock sits in its denominator while a deliberately un-gated O(N) reload probe obstructs its numerator. CPU contention alone flips it red with no engine change: four local replicates on one commit spread 0.451 to 2.49 against a 0.75 threshold. Distinct from #1096. That is the 36:00 windows-2025 step cap; this assertion failed at 30:59, beneath it. Reading the job rather than the step gives ~32 minutes and invites the wrong cause, which ci.yml already records sessions doing on this leg. Records the mechanism (the probe exempted from assertion at :182-188 is what corrupts the metric that IS asserted), the evidence that the engine was correct in the failing arm, why fd_count_monotonic cannot serve as a control, the per-message fix, a latent claim_mode grouping defect, and two plausible-but-wrong mechanisms that were retracted during the investigation. Investigation and reproduction by the session in worktree sweet-dhawan-57cbc8; independently verified here: the sweep_mode-only grouping, handles_peak as a denominator-free peak, the noisy-runner comment, the serial suite, and that 1d988fd removed the reruns=2 marker.
Collaborator
Author
|
Docs-only PR, so this is the blind mode: the drift guards live in pytest gated on All 89 skips are |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Files BACKLOG #1101. Ledger-only change - one item appended to
docs/BACKLOG.md, no code.What it records
tests/test_connscale_smoke.py:170assertsempty_claims_monotonic: the N=24 empty-claim rate per second must be at least 0.75x the N=12 rate. Wall clock is in the denominator, and a deliberately un-gated O(N) reload probe obstructs the numerator. CPU contention alone flips it red with no engine change.Same commit, same box, same config. A 0.75 threshold cannot discriminate inside a 0.451-2.49 spread.
It is not #1096, and that distinction is the point
#1096 is the 36:00 windows-2025 step cap. This assertion failed at 30:59, beneath the cap. Reading the job rather than the step gives roughly 32 minutes and invites the wrong cause - a substitution
ci.ymlalready records sessions making on this exact leg. The item states the step timing explicitly so the next reader does not merge the two stories.The sharp part
tests/test_connscale_smoke.py:182-188already exempts the reload probe from per-step assertion, in its own words "stricter than the probe's own contract and flakes on slow CI runners". Its O(N) cost - measured 0.124s at N=12 against 3.63s at N=24, longer than the entire 1.5s hold - is nevertheless loaded in full ontoempty_claims_monotonic, which is asserted. The suite declined to gate a cost and then gated a high-variance proxy for it. Disablingreload_probeunder identical contention flips the result to a pass (ratio 1.03).The engine was correct in the failing arm:
no_lossasserts at:162-164, before the SLO at:170, so every message was received, delivered and drained - and the N=24 arm was better on drain (0.801 -> 0.575s) and achieved_read (4.53 -> 5.38/s).Verification
Run against this branch:
scripts/docs/link_check.py(the repo-wide gate that landed in docs(archive): repair 270 broken archive links and add a repo-wide link-resolution gate (BACKLOG #1095) #281): 5,359 links / 347 files, all resolvetests/test_backlog_status_check.py: 19 passedIndependently verified in the code before writing, rather than taken from the investigating session's report: the
sweep_mode-only grouping atharness/load/connscale/runner.py:1084-1086,handles_peak = max(handles)at:963(a denominator-free peak, sofd_count_monotoniccannot serve as a control here), the "CI runners are noisy" comment attests/test_connscale_smoke.py:166, the serial suite viapyproject.tomladdopts, and that commit1d988fdcremoved@pytest.mark.flaky(reruns=2)under #1014.The item also records two mechanisms that were proposed and retracted during the investigation, marked as wrong, so they are not re-derived by the next reader.
Provenance
Investigation and the contention reproduction are the work of the session in worktree
sweet-dhawan-57cbc8, cited as such in the item. Filed by the coordinator session at the owner's direction. Number allocated atomically viascripts/coord/alloc.ps1, never grepped; the ledger gate passed on commit.Census note: #1101 is filed without a row in the re-scoring table, matching #1096 through #1100. The four census lines are recomputed from that table and are unchanged, so they remain accurate about it. The separate reconcile pass (open items carrying no table row) is a known outstanding job and is not attempted here.